The Facial Expression Module

In current dialogue systems the use of speech as an input modality is common. But this modality is only one of those human beings use. In human–human interaction people use gestures to point or facial expressions to show their moods as well. To give modern systems a chance to read information from all modalities used by humans, these systems must have multimodal user interfaces. The SMARTKOM system has such a multimodal interface that analyzes facial expression, speech and gesture simultaneously. Here we present the module that fulfills the task of facial expression analysis in order to identify the internal state of a user. In the following we first describe the state of the art in emotion and user state recognition by analyzing facial expressions. Next, we describe the facial expression recognition module. After that we present the experiments and results for recognition of user states. We summarize our results in the last section.